Journal of Communication and Cultural Trends (JCCT) 
Volume 3 Issue 1, Spring 2021 

ISSN@): 2706-9141 ISSN: 2706-915X 

Journal DOI: https://doi.org/10.32350/jcct 

Issue DOI: https://doi.org/10.32350/jcct.3 1 

Homepage: https://journals.umt.edu.pk/index.php/jcct/Home 





Evaluation of Learning Outcomes: A Case Study of 


Article: Secondary Level Compulsory English Textbooks of Journal QR 
Punjab Board Ore 
ae Chu 
Author(s): Amna Fayyaz', Haseeb ul Hasaan’, Kausar Parveen’ et Bro 
4, 14+ > 
i o MA 
Affiliations The University of Faisalabad, Pakistan OLS beg ar 
*Government Primary School for Boys Shareef Pura Kamoke, Pakistan 
3Govt. College Women University, Faisalabad, Pakistan Article QR 


Pub. Online: Spring 2021 





Article DOI: https://do1.org/10.32350/jcct.31.05 


Amna Fayyaz 


Article Received: Feb 9, 2021 O) 
Revised: May 26, 2021 


History: 
Accepted: May 26, 2021 TE 


Fayyaz, A., Haseeb ul Hassan & Perveen, K. (2021). ROAD = 
Evaluation of learning outcomes: A case study of 


To e secondary level compulsory English textbooks of Punjab ¢) Malica 


ance: board. Journal of Communication and Cultural Trends, nn 
3(1), 66-85. N 
Crossref f 
a 
EuroPub 
Copyright This article is open access and is distributed under the terms eres 
Information of Creative Commons Attribution 4.0 International License. 


CS 
A publication of the 
Institute of Communication and Cultural Studies 


University of Management and Technology, Lahore, Pakistan 


Evaluation of Learning Outcomes: A Study of Secondary Level Compulsory 
English Textbooks of Punjab Board, Pakistan 


Amna Fayyaz', Haseeb ul Hasaan’, Kausar Parveen’ 


'The University of Faisalabad, Pakistan 
“Government Primary School for Boys Shareef Pura Kamoke, Pakistan 
>Govt. College Women University, Faisalabad, Pakistan 


Abstract 


This study examines the learning outcomes of selected English textbooks at the 
cognitive level. The aim of the present study is to explore the levels of criticality in 
the learning outcomes of compulsory English textbooks (Grade 6-10) published by 
the Punjab Board, based on which their activities and exercises are designed. 
Learning outcomes of compulsory English textbooks (Grade 6-10) were analyzed 
following the Critical Thinking Evaluation Model (CTEM). Post-use evaluation of 
the learning outcome of compulsory English textbooks (Grade 6-10) has not been 
carried out yet. To fill this gap, selected English textbooks were evaluated in terms 
of their critical thinking skills based learning outcomes. The evaluated textbooks 
were selected through stratified sampling. A mixed-method approach including 
both qualitative and quantitative data analysis was used to make the findings more 
reliable and valid. Exploratory research design was adopted to explore the levels of 
criticality in compulsory English textbooks (Grade 6-10). The totality of results 
showed that the ‘implementation’ level of CTEM was the most frequent among 
learning outcomes. The second most frequent level of criticality was ‘background 
knowledge’ and the third most frequent was ‘apprehension’. ‘Anatomization’ and 
‘creativity’ levels of criticality appeared less frequently among the learning 
outcomes of compulsory English textbooks, while the ‘conceptualization’ level of 
CTEM was the least frequent. It was found that the learning outcomes of 
compulsory English textbooks (Grade 6-10) were not designed to enhance critical 
evaluation and self-directed learning. The results are beneficial for textbook 
designers, textbook evaluators, examiners, paper setters, teachers, students, and 
teacher trainers. This study contributes to the development of the English language 
teaching system in public schools. 


Keywords: CTEM (Critical Thinking Evaluation Model), English textbooks, 
learning outcomes 
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Introduction 


Textbooks are crucial for any sort of learning including language learning. 
Skowronski and Hinchman (2002) stated that textbooks are a major component in 
any kind of language program. Hutchinson and Torres (1994) said that the relevant 
textbooks are crucial for teaching and learning. Therefore, the learning outcomes 
of textbooks should be designed according to the needs of the students. Tomlinson 
(2011) stated that textbooks comprise an input for the learners through which they 
can practice. Textbooks play an important role in the teaching and learning process 
(Cunningsworth, 1995; Haycraft, 1998; Sheldon, 1988). Tyson (1997) stated that 
for inexperienced teachers and teachers who have no time to prepare lesson plans 
textbooks are a crucial tool; therefore, for prescribing a useful textbook evaluation 
is necessary. Learning outcomes provide a clear picture of the success of a 
particular course or program (Mahajan & Singh, 2017, p.66). Keeping in view the 
above mentioned points, this study strives to determine the levels of Critical 
Thinking Evaluation Model (CTEM) based on which the learning outcomes of 
compulsory English textbooks (Grade 6-10) have been designed. English textbooks 
taught from Grade 6 to Grade 10 were selected for analysis because these are the 
transitory levels of education whereby children move towards complex learning. 





Purpose and Objectives of the Study 


The main purpose of this study is to explore the levels of criticality in the 
learning outcomes of compulsory English textbooks (Grade 6-10). It has the 
following three objectives: 


a.To identify the levels of criticality in the learning outcomes of compulsory 
English textbooks (Grade 6-10). 

b.To explore the frequency distribution of critical thinking skills based learning 
outcomes in compulsory English textbooks (Grade 6-10). 

c.To highlight the most and least frequently used elements of ‘CTEM’ in 
compulsory English textbooks (Grade 6-10). 


Statement of the Problem 


Rea-Dickens and Germaine (1992) stated that evaluation is a process that 
highlights the appropriateness of the evaluated object. Effective learning is 
achieved with continuously developing textbooks and it needs continuous research. 
Many researches have been conducted to evaluate the textbooks’ activities, 
exercises, questions, and learning objectives (Parsaei et al., 2017, Riazi & 
Mosalanejad 2010, Naseer et al., 2020, Mahmood et al., 2020., Fayyaz, 2019). Post- 
use evaluation of the learning outcomes of compulsory English textbooks (Grade 
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6-10) has not been carried out yet in Pakistan. To fill this gap, the current research 
was conducted to highlight the levels of criticality (background knowledge, 
apprehension, conceptualization, implementation, anatomization, critical 
evaluation, creativity and self-directed learning) in the post-use learning outcomes 
of compulsory English textbooks (Grade 6-10). This study is useful for curriculum 
designers to set the learning outcomes of these textbooks appropriately and 
according to the needs of the rapidly developing era. 


Significance of the Study 


The present study will help syllabus designers to prepare more effective English 
textbooks in terms of their critical thinking skills based learning outcomes and the 
activities and exercises based on these outcomes. This research can also contribute 
in creating a balance between LOTS and HOTS-based learning outcomes of 
English textbooks. 


Research Questions 


I. What is the frequency distribution of critical thinking skills based learning 
outcomes in compulsory English textbooks (Grade 6-10)? 


a.What is the frequency of critical thinking skills based learning outcomes in 
Grade 6 English textbook? 

b.What is the frequency of critical thinking skills based learning outcomes in 
Grade 7 English textbook? 

c.What is the frequency of critical thinking skills based learning outcomes in 
Grade 8 English textbook? 

d.What is the frequency of critical thinking skills based learning outcomes in 
Grade 9 English textbook? 

e.What is the frequency of critical thinking skills based learning outcomes in 
Grade 10 English textbook”? 


2. What are the elements of Critical Thinking Evaluation Model (CTEM) used in 
determining the learning outcomes of compulsory English textbooks (Grade 6- 
10)? 

3. To what extent the learning outcomes of compulsory English textbooks (Grade 
6-10) are based on CTEM? 

Literature Review 

Theoretical Framework 


Criticality based learning outcomes of a syllabus are crucial in enabling the 
learners to make appropriate decisions, to solve real-life problems, to implement 
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educational theories in real-life situations, and to promote the society. Fayyaz 
(2019, p 6-19) proposed a model namely the Critical Thinking Evaluation Model 
(CTEM) for the evaluation of exercises, activities, learning outcomes, questions, 
and statements in textbooks for the development of curriculum, critical thinking 
skills based lesson plans, and so on. This model consists of the following eight 
levels of critical thinking skills: 


1) Background Knowledge 

2) Apprehension 

3) Conceptualization 

4) Implementation 

5) Anatomization 

6) Critical Evaluation 

7) Creativity 

8) Self-directed Learning / Independent Learning 


This model is useful to identify the levels of critical thinking skills in learning 
outcomes, questions, activities, exercises, objectives of lesson plans, objectives and 
activities of course outlines, and so on. A modified vocabulary list of CTEM is 
helpful to highlight the elements of critical thinking skills for post-use, in-use and 
pre-use evaluation of textbooks. Visual representation of CTEM 1s given below. 


Figure 1 
CTEM (Critical Thinking Evaluation Model) 
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CTEM is a developed model based on which critical thinking skills based 
learning outcomes, activities, exercises, questions, and statements in textbooks can 
be designed. 


Historical Background 


The analysis of the learning objectives of ESP books designed for the students 
of dentistry, pharmacology, and medicine taught in IUMS using Bloom’s taxonomy 
showed that there is no balance between the application of critical thinking skills 
(Parsaei et al., 2017). Riazi and Mosalanejad (2010) examined the learning 
objectives of three high school textbooks and one pre-university textbook by 
developing a coding scheme on the basis of Bloom’s taxonomy. The results 
indicated that in all grades lower-order critical thinking skills were more prevalent 
as compared to the higher-order critical thinking skills. 





Olimat (2015) evaluated the questions in Action Pack English textbook of the 
7", 8% 9" and 10“ grade using Bloom’s revised taxonomy. The results indicated 
that the level ‘background knowledge’ was found with a higher frequency in these 
textbooks. Application and synthesis level questions in the 10" grade English 
textbook were found to have a higher frequency than 7", 8" and 9" grade textbooks. 





Activities, exercises, questions, and leaning outcomes of a textbook play a 
crucial role in teaching and learning cognitive and critical thinking skills. Nakkam 
and Khamoja (2020) selected two Moroccan ELT textbooks (Ticket to English 2 
and Gateway to English 2) to analyze their reading comprehension questions using 
a checklist based on Bloom’s taxonomy. The results indicated that 77% of the total 
questions in ‘Ticket to English 2’ were designed to assess the students’ lower-order 
critical thinking skills and only 23% questions were designed to assess their higher- 
order critical thinking skills. Similarly, 84.12% of the total questions in ‘Gateway 
to English 2” were based on the lower-order critical thinking skills and only 14.78% 
questions were designed to evaluate the higher-order critical thinking skills of 
students. 





Naseer et al. (2020) evaluated the textbook of Pakistan Studies (produced by 
Punjab Textbook Board for Grade 9) using a checklist based on Bloom’s revised 
taxonomy to check the text-based questions and tasks requiring critical thinking 
skills. Text-based questions were also analyzed by categorizing them according to 
the Socratic taxonomy. The results indicated that the textbook of Pakistan Studies 
was not conducive towards enhancing the critical thinking skills and merely one 
question required the use of higher-order critical thinking skills. 
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Mahmood et al. (2020) used Bloom’s taxonomy to evaluate the exercises of 
‘Book I (English) of Federal Textbook Board and KPK Textbook Board’ of 
matriculation. The results showed that the cognitive domain was not followed 
uniformly in the selected textbook. The findings also indicated that some of the 
questions were not according to the content of unit, although the sublevels of the 
cognitive domain were incorporated in ‘Book I (English) of Federal Textbook 
Board and KPK Textbook Board’. However, they were not incorporated in a 
systematic way. 





Fayyaz (2019) evaluated the exercises and activities of compulsory English 
textbooks (Grade 6-10) using CTEM to check the various levels (background 
knowledge, apprehension, conceptualization, application, anatomization, critical 
evaluation, creativity and self-directed learning / independent learning) of critical 
thinking skills. The findings revealed that the exercises and activities of these 
textbooks were not designed to create a balance between the use of lower-order and 
higher-order critical thinking skills. Self-directed learning / independent learning 
level was totally missing in the exercises and activities of the selected textbooks. 


Textbooks are the main source of learning and teaching different skills, 
therefore, they should be designed keeping in view the learners’ needs. Badawy et 
al. (2016) conducted a study to evaluate the levels of criticality in a higher education 
textbook. The researchers used the chapter wise objectives of the selected textbook 
as sample. They analyzed the objectives of the selected textbook by following the 
list of verbs set by the Quality Management of the Institute of Statistical Studies 
and Research (ISSR), Cairo University, Egypt. The findings of their study indicated 
that the learning objectives made with synonymous verbs occurred more frequently 
than the objectives with actual verbs. 


The evolution of textbooks has become crucial for effective teaching and 
learning. Fatima et al. (2015) evaluated ‘Step Ahead 2’ and ‘Step Ahead 3’ 
textbooks used for Grade 7 and Grade 8 respectively in the Pakistani schools. A 
checklist based on McGrath’s 2002 methodology was used to make the results more 
effective. The evaluation of selected textbooks was based on the level of 
authenticity, types of activities, variety of topics, the relationship of topic with the 
learner’s interest, the material used to improve the learner’s language, grammar, 
vocabulary, types of activities, and tasks. The results showed that four basic skills 
(speaking, reading, listening, and writing) were not equally emphasized in the 
above textbooks. Cultural aspects were also not matched with the target culture. 
However, the occurrence of topics, themes, and tasks related to real-life situations 
were Satisfactory in the given textbooks. 
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Textbook is an important medium of teaching and learning, so it must be valid 
and reliable in terms of its contents, themes, tasks, activities, and exercises. 
Rahmawati (2018) evaluated the textbook ‘Primary English of a Second Language’ 
using a checklist consisting of the features of a good English textbook. The above 
textbook was analyzed for its objectives, aims, teaching and learning procedures, 
activities, exercises, and lexical choices. The results revealed that the objectives of 
the selected English textbook matched the criteria of a good English textbook, while 
the exercises in the given textbook did not match the criteria of a good English 
textbook because group or pair work exercises were absent. Vocabulary or lexical 
aspects (no new vocabulary) were also different from the criteria of a good English 
textbook. 


Research Method and Methodology 
Method and Research Design 


A mixed-method approach of data analysis was used. Qualitative data analysis 
was used to highlight the features or elements of CTEM while evaluating the 
learning outcomes of compulsory English textbooks (Grade 6-10). Quantitative 
data analysis was used to find out the frequency of critical thinking skills based 
learning outcomes of the selected textbooks. Textual analysis was conducted to 
investigate whether the learning outcomes of selected English textbooks were 
designed to enhance the critical thinking skills of students or not. 


Population and Sampling 


Stratified sampling technique was adopted to analyze the learning outcomes of 
the compulsory English textbooks (Grade 6-10) included in the sample for data 
analysis. 


Data Collection Procedure 
The process of data collection comprised the following steps: 


a) Collection of compulsory English textbooks (Grade 6-10) published by the 
Punjab Textbook Board. 

b) Conversion of the PDF files of compulsory English textbooks into MS Word 
files using an online converter. 

c) Clarification of all points of compulsory English textbooks (Grade 6-10) 
except the learning outcomes. 

d) Conversion of the MS Word files including the learning outcomes into text 
files to assess the frequency of criticality based learning outcomes through 
Antconc. 
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Instrumentation 


Critical Thinking Evaluation Model (CTEM) having eight levels including 
background knowledge, apprehension, conceptualization, implementation, 
anatomization, critical evaluation, creativity, and self-directed learning (see 
Appendix) was used to evaluate the post-use learning outcomes of compulsory 
English textbooks (Grade 6-10). Antconc 3.2.4W was used to find out the frequency 
of cognitive skills based learning outcomes. MS Excel was used to show the results 
through graphs and tables. 


Data Analysis 


Learning outcomes of compulsory English textbooks (Grade 6-10) were analyzed 
using CTEM. The results illustrate the frequency distribution of critical thinking 
skills based learning outcomes of these textbooks. It was found that the elements 
of CTEM were used to design their learning outcomes. 


Frequency Distribution of Criticality Based Learning Outcomes 


Critical thinking skills based learning outcomes of compulsory English 
textbooks (Grade 6-10) were examined and it was found that the frequency of the 
different levels of critical thinking skills varied for each textbook. The figure below 
presents the percentage of the different levels of cognitive skills used in these 
textbooks. 


Figure 2 


Presentation of Criticality Based Levels in Compulsory English 
Textbooks (Grade 6-10) 
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Grade 6 English textbook entailed a total of 123 learning outcomes. Of the total 
learning outcomes, 36.58% were at the ‘implementation’ level of criticality. This 
level appeared with the highest frequency. Learning outcomes at the level of 
‘background knowledge’ were 21.95% of the total, their percentage was 17.88% at 
the ‘anatomization’ level, 16.26% at the ‘apprehension’ level, 4.06% at the 
‘creativity’ level, and only 0.81% learning outcomes were designed to enhance the 
conceptual ability of the learners of 6" grade. Two levels of critical thinking, that 
is, ‘critical evaluation’ and “independent learning or self-directed learning’ were 
totally missing in the learning outcomes of compulsory English textbook of Grade 
6. 


Of the total learning outcomes of Grade 7 compulsory English textbook, 
35.41% were at the ‘implementation’ level of CTEM, 29.16% were at the level of 
‘background knowledge’, 18.05% were at the ‘apprehension’ level, 11.11% were 
at the ‘anatomization’ level and learning outcomes aimed to develop the creative 
ability of learners were only 6.25%. The ‘implementation’ level was the most 
frequent while the ‘creativity’ level was the least frequent. “Critical evaluation’ and 
‘independent learning or self-directed learning’ levels were absent in the learning 
outcomes of Grade 7 English textbook. 


Of the total learning outcomes of Grade 8 compulsory English textbook, 
36.78% were at the ‘implementation’ level, 22.98% were at the level of 
‘background knowledge’ and only 18.39% were intended to enhance the creative 
ability of learners. Moreover, 12.64% were at the ‘apprehension’ level, 8.04% were 
at the ‘anatomization’ level and only 1.14% of the total learning outcomes were 
about the concepts of the topics. ‘Critical evaluation’ and ‘independent learning’ or 
‘self-directed learning’ levels were absent in the learning outcomes of Grade 8 
English textbook. The ‘implementation’ level was the most frequent and the 
‘conceptualization’ level was the least frequent level in the learning outcomes of 
the above textbook. 


Among the total learning outcomes of Grade 9 compulsory English textbook, 
the most frequent level was ‘background knowledge’ with 34.54% of learning 
outcomes and the least frequent level was ‘conceptualization’ with only 1.81% of 
the total learning outcomes. Learning outcomes at the ‘implementation’ level were 
32.72%, while 14.54% were at the ‘creativity’ level, 10.99% were at the 
‘apprehension’ level, and 5.45% were at the ‘analysis’ level. The analysis of Grade 
9 compulsory English textbook yielded different results from 6", 7, and 8 grade 
English textbooks. 
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The analysis of Grade 10 compulsory English textbook showed that learning 
outcomes at the ‘apprehension’ level appeared with the highest percentage of 
35.06%, as compared to the other levels of CTEM. Learning outcomes at the 
‘implementation’ level of critical thinking skills were 29.87%, while 16.88% were 
at the ‘creativity’ level, 9.09% were at the ‘background knowledge’ level and 
3.89% were at the ‘conceptualization’ level. Moreover, learning outcomes at each 
one of the ‘anatomization’ and ‘critical evaluation’ levels were also 3.89%, while 
‘independent learning’ or “self-directed learning’ levels were completely absent in 
the learning outcomes of 10" grade compulsory English textbook. The results 
showed that the most frequent level was ‘apprehension’ and the least frequent levels 
were ‘anatomization’ and ‘conceptualization’. 


The results showed that the ‘implementation’ level of CTEM was dominant in 
the learning outcomes of compulsory English textbooks (Grade 6-10). The second 
most prominent level was “background knowledge’ and the third most prominent 
level was ‘apprehension’. The level ‘creativity’ appeared less frequently in the 
learning outcomes of all textbooks as compared to the above levels, while the levels 
‘anatomization’ and ‘conceptualization’ were the least prominent levels. The level 
‘critical evaluation’ was present only in the 10 grade English textbook with the 
minimum frequency. “Independent learning’ or ‘self-directed learning’ levels were 
totally missing from the learning outcomes of the selected textbooks. 


Elements of Criticality Used in Compulsory English Textbooks (Grade 6-10) 


The elements used to enhance or check the ability to put theory into practice 
were mostly used in compulsory English textbooks (Grade 6-10). Elements of 
‘background knowledge’ level were used with the second highest frequency in the 
learning outcomes of these textbooks. The analysis of 6, 7", and 8" grade 
compulsory English textbooks presented almost similar results. On the other hand, 
9" and 10" grade compulsory English textbooks manifested the same pattern in 
terms of criticality. The results indicated that different elements or action verbs of 
CTEM were used in the various textbooks. The table below presents the list of 
elements used in these textbooks. 


The table shows that the selected textbooks varied significantly from each other 
in terms of critical thinking skills based learning outcomes. Grade 7 compulsory 
English textbook had the minimum number of the elements of criticality in its 
learning outcomes, while Grade 6 compulsory English textbook had more learning 
outcomes with criticality based elements as compared to any other compulsory 
English textbook. Some of the elements were found to be similar in various 
textbooks such as ‘recognize, identity, write, answer, illustrate, differentiate, make, 
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analyze, demonstrate, and answer’. Moreover, 24 elements of criticality were used 
in the Grade 6 compulsory English textbook, 16 elements of criticality were used 
in the Grade 7 compulsory English textbook, 21 elements of criticality were used 
in the Grade 8 compulsory English textbook, 18 elements of criticality were used 
in the Grade 9 compulsory English textbook, and 23 elements of criticality were 


used in the Grade 10 compulsory English textbook. 


Table 1 


List of elements used in various compulsory English textbooks (Grade 6-10) 


6 


Recognize 
Identify 
Show 
Write 
Answer 
Illustrate 
Express 
Summarize 
Compare 
Demonstrate 
Distinguish 
Locate 
Interpret 
Relate 
Apply 
Develop 
Use 
Support 
Analyze 
Classify 
Examine 
Differentiate 
Plan 

Make 
Write 
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7 


Recognize 
Identify 
Write 
Answer 
Illustrate 
Express 
Summarize 
Demonstrate 
Discuss 
Locate 
Apply 
Find 

Use 
Support 
Analyze 
Plan 
Develop 
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8 


Recognize 
Identify 
Recite 
Describe 
read 
Retell 
Express 
Choose 
Demonstrate 
Respond 
Deduce 
Develop 
Utilize 
Change 
Use 
Transfer 
Analyze 
Infer 
Propose 
Differentiate 
Make 
Write 


9 


Recognize 
Identify 
Describe 
Read 
Paraphrase 
Demonstrate 
Respond 
Interpret 
Relate 
Develop 
Utilize 
Practice 
Use 
Analyze 
Organize 
Plan 
Construct 
Organize 
Write 


10 


Recognize 
Identify 
State 

Fill 
Illustrate 
Summarize 
Explain 
Paraphrase 
Demonstrate 
Determine 
Locate 
Deduce 
Change 
Practice 
Use 
Produce 
Support 
Analyze 
Differentiate 
Evaluate 
Support 
Make 
Synthesize 
Write 
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Representation of CTEM in the Learning Outcomes of Compulsory English 
Textbooks (Grade 6-10) 


The overall analysis of the selected textbooks highlighted the ‘implementation’ 
level of CTEM as the most frequent in their learning outcomes. The ‘background 
knowledge’ level was ranked as the second most frequent level and the third most 
frequent level was ‘apprehension’. ‘Anatomization’ and ‘creativity’ levels of 
CTEM were found to be less frequent as compared to the above levels. The least 
frequent levels of CTEM were ‘conceptualization’ and ‘critical evaluation’ in all 
textbooks. The figure below shows the total results of the critical thinking skills 
based learning outcomes of compulsory English textbooks (Grade 6-10). 


Figure 3 
Representation of CTEM levels in compulsory English textbooks (Grade 6-10) 
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Total results depicting the distribution of the levels of critical thinking skills in 
the learning outcomes of compulsory English textbooks (Grade 6-10) showed that 
the ‘implementation’ level of CTEM was the most frequent with 172 learning 
outcomes. The level “background knowledge’ appeared with a frequency of 115 
learning outcomes, while the ‘apprehension’ level appeared 90 times in the learning 
outcomes of the selected textbooks. The frequency of learning outcomes at the 
‘creativity’ level was 51 and their frequency at the ‘anatomization’ level was 50, 
respectively. The ‘conceptualization’ level appeared with a frequency of 6 learning 
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outcomes and the level ‘critical evaluation’ appeared only in the Grade 10 
compulsory English textbook. 


Discussion 


Richards (2014) stated that post-use evaluation provides information which helps 
to decide whether the evaluated object or book is useful or not. Based on the 
evaluation, some changes are made in the evaluated object or book because 
curriculum, syllabus, and textbooks need to be revised and updated in order to fulfill 
the requirements of the rapidly developing science, technology, and trends across 
the world. These changes result not only in the revision of the contents of the 
textbooks but also of their presentation (Mahmood, 2009, p.158). The present study 
examined the learning outcomes of compulsory English textbooks (Grade 6-10) to 
highlight the levels of criticality in them because the process of teaching and 
learning relies heavily on these outcomes. Tikhonova and Kudinova (2015) stated 
that sophisticated thinking is a balance between well-developed lower-order and 
higher-order thinking skills. However, the findings of the current study showed that 
the ‘implementation’ level of cognitive skills was most prominent in the learning 
outcomes of the selected textbooks, which shows that lower-order thinking skills 
based learning outcomes were the focus of syllabus designers. Fayyaz (2019) 
introduced eight levels of critical thinking skills within the Critical Thinking 
Evaluation Model (CTEM) for the step by step development of the cognitive skills 
of learners through textbooks. Syllabus designers must focus to set the learning 
outcomes by following the features of each level of critical thinking skills. 
However, this research found that the “self-directed learning’ level of CTEM was 
totally missing from the learning outcomes of compulsory English textbooks 
(Grade 6-10). 


Conclusion 











The findings of the given research showed that the compulsory English 
textbooks (Grade 6-10) need to be improved with respect to criticality-based 
learning outcomes in order to enhance the critical thinking skills of the learners. It 
was found that different compulsory English textbooks encompassed different 
levels of criticality-based skills. Moreover, it was also found that the selected 
textbooks were not designed for the most part to promote higher-order critical 
thinking skills. Based on the above findings, it is concluded here that syllabus 
designers must be trained to set higher-order critical thinking skills based learning 
outcomes for designing English textbooks in order to promote higher-order critical 
thinking skills of the learners through these textbooks. 
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Recommendations 


For the development of textbooks and to facilitate the process of teaching, 
learning, and assessment, post-use evaluation of the learning outcomes of course 
outlines, lesson plans, textbooks, and examination questions should be carried out 
to explore the levels of criticality. CTEM is useful for the pre-use, in-use and post- 
use evaluation of textbooks, syllabus, curriculum, examination papers, and other 
teaching and learning tasks in order to evaluate their criticality. 


Implications of the Study 


The current study is useful for syllabus designers to develop the selected 
textbooks further for teaching and learning criticality through them. 
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Appendix 


Critical Thinking Evaluation Model (CTEM) was introduced in the thesis of Master 
of Philosophy in Linguistics titled “Evaluation of Exercises and Activities of 
English Textbooks at Secondary Level”. The thesis was written in Riphah 
International University Faisalabad. This model is used to evaluate the exercises, 
learning outcomes, and activities of textbooks. A description of Critical Thinking 
Evaluation Model (CTEM) including the description and imperative verbs of each 
one of its levels is given below. 


Level 1: Background Knowledge 


Gaining information about a particular object is the fundamental level of critical 
thinking skills based learning outcomes. This level of critical thinking skills 
involves command, grip, education, proficiency, schooling, skill, accomplishment, 
enlightenment, and philosophy regarding a topic. Command on this level is crucial 
to move to the next level. Following is a list of imperative verbs to set and evaluate 
exercises, activities, objectives, questions, lesson plans, course outlines, and 
examination papers. 


define identify recall describe order outline 
read list list the name recite show reproduce 
point answer recognize memorize underline state 

cite write circle label remember record 
repeat match quote 


Level 2: Apprehension 


Apprehension is the next level and it aims to acquire information or knowledge 
about a particular subject. This level requires understanding, perception, 
realization, cognition, and discernment about the subject area and its nature. 
Imperative verbs used to assess the apprehension skills of students are as follows. 


reorder express explain compare demonstrate indicate 
tell convert illustrate Summarize paraphrase rephrase 
discuss locate review defend translate contrast 
extend estimate distinguish report interpret retell 
restate generalize choose determine respond select 


Level 3: Conceptualization 


Under this level of critical thinking skills, learners invent new ideas using their 
command over background knowledge and apprehension (the two previous levels). 
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Concepts are vital to recreate, assemble, and rethink about something. Verbs which 
present the conceptual ability of students are listed below. 


recreate reason provoke represent think over resolve 
assemble precaution consider relate gather passionate 
deduce compare 


Level 4: Implementation 


At this level of critical thinking skills, students become capable of applying 
knowledge, apprehension and concepts to sort out real-life problems and make their 
daily life activities more effective. At this level, students are able to use educational 
approaches, principles, theories, and procedures to deal with daily life activities and 
exercises. ‘Implementation’ level 1s evaluated using the following list of verbs. 


apply divide change find use solve 
generalize develop act out select complete utilize 
calculate schedule rewrite add practice manipulate 
show sketch support transfer verify prove 
operate employ 


Level 5: Anatomization 


Knowledge, understanding, concept and implementation enable students to 
examine, analyze, break down, interpret, integrate, and draw conclusions about a 
subject. Verbs which show the ability of anatomization are enlisted below. 


analyze breakdown categorize classify conclude deduce 
discriminate diagram distinguish examine infer inspect 
investigate order survey modify compose verify 
hypothesize separate propose act out formulate value 
blend unite relate connect differentiate plan 
generalize organize subdivide initiate detect 


Level 6: Critical Evaluation 


Critical evaluation is helpful to judge, decide, argue, assess, and criticize by 
following a specific criterion. Following are the codes or verbs used to evaluate the 
‘critical evaluation’ level of critical thinking skills. 


appraise debate judge choose standardize measure 
prioritize support attach contrast test select 
rate criticize 
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Level 7: Creativity 


Creativity enables students to create, originate, and integrate something new by 
utilizing background knowledge, apprehension, conceptualization, 
implementation, anatomization, and critical evaluation of the subject. Verbs which 
are useful to examine the creativity in a statement are listed below. 


create construct manage reorganize systematize plan 
combine collect compose invent arrange design 
originate develop synthesize make relate device 
formulate organize modify predict reconstruct propose 
rearrange 


Level 8: Independent Learning / Self-directed Learning 


By mastering all the above levels students become self-directed learners. At this 
stage, students inquire about the particular subject on their own. ‘Independent 
learning’ or “self-directed learning’ is analyzed using the following verbs. 


analyze transfer self- self- take self-assess 
knowledge monitor regulate responsibility 

reflect predict set goals research generalize formulate 
performance knowledge 

inquire frame develop a identify create a plan construct 
questions learning needs knowledge 

plan 
describe 
how to 
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